Image processing method, image processor, integrated circuit, and program

ABSTRACT

An image processing method capable of appropriately increasing a resolution of an input image in which an edge direction that is a direction along an edge included in the input image is identified; a shape of an application region that is a region including at least a part of the edge is determined according to the identified edge direction; an image similar to an image within the application region having the determined shape is searched for; and an output image is generated by performing a resolution conversion process on the input image using the similar image so that the input image includes high-frequency components.

CROSS REFERENCE TO RELATED APPLICATION

The present application is based on and claims priority of JapanesePatent Application No. 2011-094465 filed on Apr. 20, 2011. The entiredisclosure of the above-identified application, including thespecification, drawings and claims is incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present invention relates to an image processing method, an imageprocessor, an integrated circuit, and a program for generating, using aninput image, an output image having a resolution higher than aresolution of the input image.

BACKGROUND OF THE INVENTION Background Art

When a resolution of an image to be displayed on a high resolutiondisplay is lacking, it is necessary to enlarge the image so as to matchthe resolution of the image to a resolution of the high resolutiondisplay.

Although methods of increasing the resolution of an image have beenconventionally suggested, there is a limit on processing capacity forthe practical use. Thus, the combination of simple image enlargement byinterpolation and image enhancement processing has supported themethods. Thus, problems of image degradation have occurred, such asblurring and noticeable jaggies in an edge.

Recent technological advances in performance of hardware have enabledallocation of larger processing capacity to the image enlargementprocessing. Here, attention is currently focused on the super-resolutiontechnique capable of converting an image to an image with high qualityand high resolution through complex processing.

Among the image processing methods using the super-resolution technique,a method using a training database storing data learned from examples ofcorrespondence between high-resolution images and low-resolution imagesis called a training-based super-resolution (see PTL 1).

CITATION LIST Patent Literature

-   [PTL 1] Japanese Unexamined Patent Application Publication No.    2003-018398

SUMMARY OF INVENTION

However, the image processing method of PTL 1 has a problem that theresolution of an input image cannot be appropriately increased.

Thus, the object of the present disclosure is, in view of the problem,to provide an image processing method of appropriately increasing theresolution of an input image.

In order to achieve the object, the image processing method according toan aspect of the present invention is an image processing method ofgenerating, using an input image, an output image having a resolutionhigher than a resolution of the input image, and the method includes:identifying an edge direction that is a direction along an edge includedin the input image; determining a shape of an application regionaccording to the identified edge direction, the application region beinga region including at least a part of the edge; searching for an imagesimilar to an image within the application region having the determinedshape; and generating the output image by performing a resolutionconversion process on the input image using the similar image so thatthe input image includes high-frequency components.

The general or specific aspects may be implemented by a system, anapparatus, an integrated circuit, a computer program, or a recordingmedium, or by an arbitrary combination of the system, the apparatus, theintegrated circuit, the computer program, and the recording medium.

With the image processing method according to the present disclosure,the resolution of an input image can be appropriately increased.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present invention. In the Drawings:

FIG. 1 is a block diagram illustrating a configuration of atraining-based image processor;

FIG. 2 is a flowchart of the image processing performed by thetraining-based image processor;

FIG. 3 is a block diagram illustrating a configuration of an imageprocessor according to Embodiment 1 of the present invention;

FIG. 4 illustrates data stored in a training database according toEmbodiment 1;

FIG. 5 is a block diagram illustrating a configuration of a trainingimage generating apparatus according to Embodiment 1;

FIG. 6 is a flowchart of the image processing performed by the imageprocessor according to Embodiment 1;

FIG. 7 illustrates an image processing method according to Embodiment 1;

FIG. 8 illustrates a temporary enlarged image including blocks accordingto Embodiment 1;

FIG. 9A illustrates a horizontal Sobel filter according to Embodiment 1;

FIG. 9B illustrates a vertical Sobel filter according to Embodiment 1;

FIG. 10A illustrates an application region in a vertical edge directionaccording to Embodiment 1;

FIG. 10B illustrates an application region in a horizontal edgedirection according to Embodiment 1;

FIG. 11 illustrates processing performed by the training data searchunit according to Embodiment 1;

FIG. 12 is a block diagram illustrating a configuration of an imageprocessor according to Embodiment 2 of the present invention;

FIG. 13 is a flowchart of the image processing performed by the imageprocessor according to Embodiment 2;

FIG. 14 illustrates a position matching process according to Embodiment2;

FIG. 15A illustrates an example of a physical format of a flexible diskaccording to Embodiment 3 of the present invention;

FIG. 15B illustrates a front appearance of the flexible disk, a crosssection of the flexible disk, and the flexible disk according toEmbodiment 3;

FIG. 15C illustrates a configuration for recording a program on aflexible disk and reproducing the program according to Embodiment 3;

FIG. 16 illustrates a configuration of a television receiver accordingto Embodiment 4 of the present invention; and

FIG. 17 is a block diagram illustrating a functional configuration of animage processor according to another aspect of the present invention.

DETAILED DESCRIPTION OF THE INVENTION Knowledge on which the PresentInvention is Based

The present inventors gave an example of an a training-based imageprocessor, regarding the image processing method disclosed in PTL 1, andfound a problem in the example.

FIG. 1 is a block diagram illustrating a configuration of thetraining-based image processor as the example.

As illustrated in FIG. 1, an image processor 600 includes a trainingdatabase 610, an image enlargement unit 605, a search vector generatingunit 620, a training data search unit 630, and an addition unit 680.

The image processor 600 converts an input image 601 of a low resolutioninto an output image 602 of a high resolution, using the trainingdatabase 610 that stores index vectors and high-frequency component dataitems 631 that are associated one-to-one with the index vectors. Each ofthe index vectors is an image of low-frequency components.

The training database 610 stores the index vectors and thehigh-frequency component data items 631 that are associated one-to-onewith the index vectors. Each of the index vectors and the high-frequencycomponent data items 631 is a block-shaped image.

FIG. 2 is a flowchart of the image processing performed by the imageprocessor 600.

At Step S4001, the image enlargement unit 605 generates a temporaryenlarged image 606 by enlarging the input image 601.

The following processes are performed on each current block to beprocessed included in the temporary enlarged image 606 that is anenlarged image of the input image 601.

At Step S4002, the search vector generating unit 620 generates a searchvector 621 by extracting, from the temporary enlarged image 606, amedium-frequency component data item and a high-frequency component dataitem in a region including the current block. The region has the sameshape as that of the current block. In other words, the search vector621 is a block-shaped image including the medium-frequency componentdata item in a current block to be processed, and the high-frequencycomponent data item in a part of a region of a block adjacent to thecurrent block.

At Step S4003, the training data search unit 630 selects one of thehigh-frequency component data items 631 that corresponds to an indexvector having the highest similarity to the search vector 621, fromamong the index vectors stored in the training database 610.

At Step S4004, the addition unit 680 adds the selected high-frequencycomponent data item 631 to the current block of the temporary enlargedimage 606. With iterations of the processes from Steps S4002 to S4004for each of the blocks, the output image 602 of the high resolution isgenerated.

With the training-based image processor and the image processing methodthat are given as the example, the search vector 621 is calculated froma region having the same shape as that of the current block. In otherwords, when the current block is a square, the search vector 621 iscalculated from a square region. Since a ratio of a length of a diagonalline of the square to a length of a side of the square is a square rootof 2 to 1, the side horizontal or vertical to the square region isshorter than the diagonal line, and the edge features in the horizontalor vertical direction that are included in the region tend to be smallerthan the edge features in the diagonal direction along the diagonalline. In other words, even when the edge features in the region alongthe diagonal line can sufficiently be obtained, the edge features in thehorizontal or vertical direction cannot sufficiently be obtained.

Thus, with the training-based image processor and the image processingmethod that are given as the example, it is not possible to search foran appropriate index vector having similarities with the features in theregion including the edge in the horizontal or vertical direction, andto sufficiently increase the resolution of the image of the blockincluded in the region. As a result, the super-resolution effect(precision of increasing a resolution) for each block becomesnon-uniform depending on a direction of an edge included in the regioncorresponding to the block.

The present invention has been conceived to solve the problems, and hasan object of providing an image processing method, an image processor,and others for appropriately increasing the resolution of an input imageby obtaining the super-resolution effect equivalent in any edgedirection.

In order to achieve the object, the image processing method according toan aspect of the present invention is an image processing method ofgenerating, using an input image, an output image having a resolutionhigher than a resolution of the input image, and the method includes:identifying an edge direction that is a direction along an edge includedin the input image; determining a shape of an application regionaccording to the identified edge direction, the application region beinga region including at least a part of the edge; searching for an imagesimilar to an image within the application region having the determinedshape; and generating the output image by performing a resolutionconversion process on the input image using the similar image so thatthe input image includes high-frequency components.

Since the shape of the application region is not fixed and is determinedaccording to an edge direction, an image in the application region inany direction of an edge can be used. The application regionsufficiently includes features of the edge. Furthermore, since a similarimage is searched for and the resolution conversion process isperformed, using the image in the application region having thedetermined shape, the super-resolution effect can be obtainedequivalently for an edge in any direction. As a result, it is possibleto suppress non-uniformity in the super-resolution effect for eachblock, and appropriately increase the resolution of the input image.

For example, in the determining, a shape having a second width and afirst width longer than the second width may be determined as the shapeof the application region, the second width being a width of theapplication region in the identified edge direction, and the first widthbeing a width of the application region in a direction vertical to theedge direction.

As such, since the input image includes more features in the directionvertical to the edge direction than features in the edge direction, thesuper-resolution effect with higher precision can be obtained for anedge in any direction by determining the shape having the second widthin the edge direction and the first width that is longer than the secondwidth and is in the direction vertical to the edge direction, as theshape of the application region.

Furthermore, for example, in the identifying, an edge direction of anedge may be identified for each of blocks included in the input image,the edge being included in the block, in the determining, a shape of anapplication region may be determined for each of the blocks, accordingto the edge direction identified for the block, in the searching, animage similar to an image within the application region having the shapedetermined for the block may be searched for each of the blocks, and inthe generating, the output image may be generated by performing theresolution conversion process on each of the blocks using the similarimage that is searched for the block.

Accordingly, the equivalent super-resolution effect can be obtained foreach of the blocks.

Furthermore, for example, in the determining, a shape having the firstwidth and the second width may be determined for each of the blocks asthe shape of the application region for the block, the first width beinglonger than a width of the block in a direction vertical to the edgedirection, the second width being shorter than a width of the block inthe edge direction, and the edge direction being identified for theblock. More specifically, in the determining, the shape of theapplication region may be determined so that the number of pixelsincluded in the application region corresponding to each of the blocksis equal to or smaller than the number of pixels included in the block.

Accordingly, since the application region has a size identical to orsmaller than the block corresponding to the application region, thesearching load can be more reduced than the case of searching for asimilar image that is similar to the image in the block.

Furthermore, for example, in the searching, (i) a database may be used,the database holding a plurality of search images, and a plurality ofapplication images associated one-to-one with the search images andincluding more high-frequency components than high-frequency componentsof the search images, and (ii) the search images held in the databasemay be searched for, as the similar image, a search image including animage similar to an image within an application region having a shapedetermined for a current block to be processed, and in the generating,the resolution conversion process may be performed on the current blockby selecting, from the database using the similar image obtained in thesearching, an application image associated with the similar image andadding the selected application image to the current block.

Accordingly, the resolution conversion process is performed on thecurrent block by adding, to the current block, the application image(the training high-frequency image) held in the database associated withthe search image that is similar to the image within the applicationregion. Thus, the resolution of the current block can be simplyincreased by appropriately managing the associations in the database.

Furthermore, for example, in the searching, the database may be searchedfor a plurality of search images as a plurality of similar images, thesearch images each including an image similar to the image within theapplication region having the shape determined for the current block,and in the generating, the resolution conversion process may beperformed on the current block by (i) selecting, from the database usingthe similar images obtained in the searching, a plurality of applicationimages associated one-to-one with the similar images, (ii) generating asynthesized image by synthesizing the selected application images, and(iii) adding the synthesized image to the current block.

After synthesizing a plurality of application images, the synthesizedimage is added to the current block. Thus, noise included in the outputimage can be reduced.

Furthermore, for example, a plurality of application images larger insize than the blocks may be held in the database in one-to-oneassociation with the search images, and in the generating, thesynthesized image may be added to the current block and a part of atleast one of the blocks around the current block so that the center ofthe synthesized image matches the center of the current block, thesynthesized image being larger than the current block.

Since the application images and the synthesized image are larger thanthe current block, addition is performed so that the center of thesynthesized image matches the center of the current block. Thus,artifacts between the blocks can be reduced.

Furthermore, for example, the image processing method may furtherinclude enlarging the input image, wherein the identifying, thedetermining, the searching, and the generating may be performed on eachof blocks included in the enlarged input image.

Accordingly, the resolution of the input image can be appropriatelyincreased.

Furthermore, for example, the image processing method may furtherinclude extracting, as medium-frequency components from each of theblocks included in the enlarged input image, frequency componentsexcluding low-frequency components, wherein in the searching, a searchimage may be searched for each of the blocks, the search image includingan image similar to an image of the medium-frequency components withinan application region having a shape determined for the block.

Since the images of medium-frequency components included in the blocksof the input image are used for search, the search images in thedatabase can be images only of medium-frequency components, and theapplication images associated with the search images can be images ofhigh-frequency components. Furthermore, since the search images do notinclude low-frequency components, the correlation between the searchimages and the application images can be increased. As a result, theresolution of the input image can be appropriately increased.

Furthermore, for example, in the searching, a similar imagecorresponding to a current block to be processed may be searched for bymatching a position of the image within the application region havingthe determined shape to a position of a picture included in a movingimage, the picture being other than the input image, and the movingimage including the input image that is a picture, and in thegenerating, a reference block including at least a part of the similarimage obtained in the searching may be obtained from the picture otherthan the input image using the similar image, and the resolutionconversion process may be performed on the current block byinterpolating pixels in the current block using pixels in the referenceblock.

The similar images for the current block are searched for by matchingthe position of the picture other than the input image to the positionof the image within the application region with the sub-pixel precision,that is, by performing the motion estimation and the motion compensationwith the sub-pixel precision. Accordingly, the resolution of the movingimage can be appropriately increased.

The image processing method and the image processor according to one ormore aspects of the present invention will be specifically describedwith reference to drawings.

Each Embodiment below describes a specific example of the presentinvention. The values, shapes, materials, constituent elements,dispositions and connections of the constituent elements, steps, and theorder of the steps are examples, and do not limit the present invention.Furthermore, among the constituent elements in each Embodiment, theconstituent elements that are not described in independent claimsindicating the most generic concept are described as arbitraryconstituent elements.

Embodiment 1

FIG. 3 is a block diagram illustrating a configuration of an imageprocessor 100 according to Embodiment 1 in the present invention. Theimage processor 100 is an apparatus that appropriately increase theresolution of an input image, and generates, using an input image 101received from outside of the image processor 100, an output image 102having a resolution higher than that of the input image 101. The inputimage 101 may be any one of a moving image and a still image.

As illustrated in FIG. 3, the image processor 100 includes a trainingdatabase 110, an image enlargement unit 120, an edge directioncalculating unit 130, a region calculating unit 140, a feature dataextracting unit 150, a training data search unit 160, a synthesizingunit 170, and an addition unit 180.

First, the training database 110 will be described.

FIG. 4 illustrates data stored in the training database 110.

As illustrated in FIG. 4, the training database 110 stores traininghigh-frequency images P11[1], P11[2], . . . , P11[K], and trainingmedium-frequency images P12[1], P12[2], . . . , P12[K]. Here, K is aninteger equal to or larger than 2.

Hereinafter, each of the training high-frequency images P11[1], P11[2],. . . , P11[K] is simply referred to as a training high-frequency imageP11. Furthermore, each of the training medium-frequency images P12[1],P12[2], . . . , P12[K] is simply referred to as a trainingmedium-frequency image P12. The training medium-frequency image P12 is asearch image, and the training high-frequency image is an applicationimage.

The training database 110 stores training image pairs of the traininghigh-frequency images P11 and the training medium-frequency images P12that are associated with each other. More specifically, the trainingdatabase 110 stores the training image pairs P10[1], P10[2], . . . ,P10[K]. In other words, the training database 110 holds (i) a pluralityof search images (the training medium-frequency images P12), and (ii) aplurality of application images (the training high-frequency images P11)that are associated one-to-one with the search images and include morehigh-frequency components than those of the search images.

Hereinafter, each of the training image pairs P10[1], P10[2], . . . ,P10[K] is also simply referred to as a training image pair P10. The Ktraining image pairs P10 are different from each other. Each of thetraining image pairs P10 is a training image pair in which the traininghigh-frequency image P11 and the training medium-frequency image P12that are generated from the same block in the same image are associatedwith each other.

Next, a method of generating the training high-frequency image P11 andthe training medium-frequency image P12 will be described. The traininghigh-frequency image P11 and the training medium-frequency image P12 aregenerated by a training image generating apparatus 50 to be describedhereinafter.

FIG. 5 is a block diagram illustrating a configuration of the trainingimage generating apparatus 50. FIG. 5 also illustrates the trainingdatabase 110 included in the image processor 100 for describing thetraining image generating apparatus 50. The training image generatingapparatus 50 generates the training high-frequency images P11 and thetraining medium-frequency images P12, using each of training images P1captured in advance by, for example, a digital camera. The trainingimage generating apparatus 50 processes several hundred training imagesP1, for example.

As illustrated in FIG. 5, the training image generating apparatus 50includes a low-pass filter unit 51, a 1/N reduction unit 52, an Nenlargement unit 53, a high-pass filter unit 54, and a high-frequencycomponent extracting unit 55.

The high-frequency component extracting unit 55 extracts a differencebetween the training image P1 and a training low-frequency image P4 thatis transmitted from the N enlargement unit 53 and is to be describedlater, as high-frequency components of the training image P1, andpartitions an image including the high-frequency components into blocksof a predetermined size. Then, the high-frequency component extractingunit 55 stores each of the resulting blocks in the training database 110as the training high-frequency image P11.

The high-frequency component extracting unit 55 may extracthigh-frequency components of the training image P1 by linear filteringand others.

Furthermore, the low-pass filter unit 51 extracts low-frequencycomponents of the training image P1 as a training low-frequency image P2by linear filtering and others.

The 1/N reduction unit 52 reduces the training low-frequency image P2 to1/N in each of the horizontal and vertical directions to generate atraining low-frequency image P3.

The N enlargement unit 53 enlarges the training low-resolution image P3by a factor N in each of the horizontal and vertical directions togenerate the training low-frequency image P4. The N enlargement unit 53transmits the training low-frequency image P4 to the high-pass filterunit 54 and the high-frequency component extracting unit 55.

The high-pass filter unit 54 extracts high-frequency components from thetraining low-frequency image P4 by linear filtering and others, andpartitions an image of the high-frequency components into blocks of afixed size. Then, the high-pass filter unit 54 stores each of theresulting blocks in the training database 110 as the trainingmedium-frequency image P12.

The training image generating apparatus 50 may generate the trainingmedium-frequency image P12 only through processes performed by thelow-pass filter unit 51 and the high-pass filter unit 54, without usingthe 1/N reduction unit 52 and the N enlargement unit 53.

The size of the training high-frequency image P11 is the same as that ofthe training medium-frequency image P12. Each of the traininghigh-frequency image P11 and the training medium-frequency image P12has, for example, a size of horizontal 18×vertical 18 pixels.

Here, the high-pass filter unit 54 and the high-frequency componentextracting unit 55 store, in the training database 110, the trainingimage pair P10 in which the training high-frequency image P11 and thetraining medium-frequency image P12 correspond to a block at the samecoordinates (position) in the same training image P1 and are associatedwith each other, as illustrated in FIG. 4.

The process of storing the training image pair P10 in the trainingdatabase 110 is performed on all of the target training images P1 to beprocessed, so that K training image pairs P10 are stored in the trainingdatabase 110. K is, for example, 100,000. In other words, for example,the 100,000 training image pairs P10 are stored in the training database110.

Here, the training high-frequency image P11 and the trainingmedium-frequency image P12 corresponding to each of the training imagepairs P10 are images corresponding to a block at the same coordinates(position) in the same training image P1.

With the processes, a lot of kinds of the training image pairs P10 arestored in the training database 110.

The training high-frequency image P11 is a synthesized image 171 to bedescribed later, and is a training data item for generating an outputimage of a high resolution. In other words, the training database 110stores K pairs of training data items. In each of the pairs, the firsttraining data item (training high-frequency image P11) obtained from thehigh-frequency components of the training image P1 is associated withthe second training data item (training medium-frequency image P12)obtained at least from the low-frequency components of the trainingimage P1.

The image enlargement unit 120 generates a temporary enlarged image 121by enlarging the input image 101. The input image 101 is an image havinga resolution to be converted. The input image 101 has, for example, asize of horizontal 1920 pixels×vertical 1080 pixels. The size of theinput image 101 is not limited to the size of horizontal 1920pixels×vertical 1080 pixels, and the input image 101 may have, forexample, a size of horizontal 1440 pixels×vertical 1080 pixels.

The edge direction calculating unit 130 extracts horizontal feature dataand vertical feature data from the enlarged input image 101 (temporaryenlarged image 121), and calculates an edge direction 131 based on thehorizontal feature data and the vertical feature data. The details willbe described later. The edge direction 131 is a direction along an edge,and indicates an angle of the edge. More specifically, the edgedirection 131 represents a vertical angle with respect to an arraydirection of pixels that (i) are included in a block to be processed(hereinafter referred to as a current block to be processed) within thetemporary enlarged image 121 and (ii) have larger change in theluminance. In other words, the edge direction calculating unit 130identifies, for each of the blocks included in the temporary enlargedimage 121 that is the enlarged input image 101, an edge direction thatis a direction along an edge included in the block.

The region calculating unit 140 calculates, within the temporaryenlarged image 121 based on the edge direction 131, an applicationregion to be used for extracting region feature data 151 for calculatinga similarity with the training medium-frequency image P12. The detailswill be described later. In other words, the region calculating unit 140determines, for each of the blocks, the shape of the application regionthat is a region including at least a part of the edge in the block,according to the edge direction identified for the block.

The feature data extracting unit 150 extracts the region feature data151 from the application region. The region feature data 151 is an imageof medium-frequency components. In other words, the feature dataextracting unit 150 extracts, for each of the blocks included in thetemporary enlarged image 121 that is the enlarged input image 101,frequency components excluding the low-frequency components as theregion feature data 151.

The training data search unit 160 calculates a similarity between theregion feature data 151 and each of the K training medium-frequencyimages P12 stored in the training database 110. The details will bedescribed later. Then, the training data search unit 160 selects thetraining high-frequency images P11 corresponding one-to-one to thetraining medium-frequency images P12, based on the calculatedsimilarities. In other words, the training data search unit 160searches, using the training database 110, K search images held in thetraining database 110 (training medium-frequency images P12) for aplurality of search images each including an image within an applicationregion having the shape determined for the current block, as respectivesimilar images. Then, the training database 110 selects applicationimages (training high-frequency images P11) associated one-to-one withthe similar images (training medium-frequency images P12) in thetraining database 110, using the similar images.

The synthesizing unit 170 generates the synthesized image 171 using theselected training high-frequency images P11. The details will bedescribed later. In other words, the synthesizing unit 170 generates thesynthesized image 171 using the selected application images (traininghigh-frequency images P11).

The addition unit 180 adds the synthesized image 171 to the temporaryenlarged image 121 per block to generate the output image 102. Thedetails will be described later. In other words, the adding unit 180adds the synthesized image 171 to the current block so as to perform aresolution conversion process on the current block. The adding unit 180performs such addition per block to generate the output image 102.

According to the image processor in FIG. 1 and the image processingmethod in FIG. 2, one of the high-frequency component data items isselected using the feature data (search vector) extracted from a regionhaving the same shape as that of the current block. On the other hand,in Embodiment 1, the training high-frequency images P11 are selectedusing the region feature data 151 extracted from a region from which theedge features are easily obtained, according to the edge direction.

Next, the image processing method performed by the image processor 100will be specifically described with reference to FIGS. 6 to 10.

FIG. 6 is a flowchart of the image processing performed by the imageprocessor 100 according to Embodiment 1. Embodiment 1 is describedassuming that the input image 101 is a still image.

The image processor 100 converts the input image 101 into the outputimage 102 that is a high-resolution image, using an image stored in thetraining database 110.

At Step S1001, the image enlargement unit 120 enlarges the input image101 by a factor N in each of the horizontal and vertical directions togenerate the temporary enlarged image 121 as illustrated in FIG. 7,where N is a real number larger than 1. The process of enlarging theinput image 101 uses, for example, bicubic interpolation. Embodiment 1assumes that the input image 101 is enlarged by a factor of 2 in each ofthe horizontal and vertical directions, for example.

In other words, Step S1001 is a step of enlarging the input image 101 togenerate an enlarged image (temporary enlarged image 121).

The technique for enlarging the input image 101 is not limited to thebicubic interpolation, and a pixel interpolation method such as splineinterpolation may be used. Furthermore, enlargement factors in thehorizontal and vertical directions may be different from each other. Forexample, when an image of 720×480 pixels are enlarged to an image of1920×1080 pixels, the enlargement factor of the input image in thehorizontal direction is different from the enlargement factor thereof inthe vertical direction.

The process of enlarging an image is not limited to enlarging the imageto be larger than the input image 101. For example, the input image 101may be enlarged by a factor N that is equal to or smaller than 1.

As illustrated in FIG. 8, the image enlargement unit 120 partitions thetemporary enlarged image 121 into blocks NB. The blocks NB are a matrixof m rows and n columns. Hereinafter, the block NB at the m-th row andthe n-th column in the temporary enlarged image 121 will be denoted as ablock NB [mn]. For example, the block NB at the first row and the secondcolumn in the temporary enlarged image 121 will be denoted as a block NB[12].

The block NB is smaller than the training high-frequency image P11 andthe training medium-frequency image P12. The block NB has, for example,a size of horizontal 12 pixels×vertical 12 pixels.

Here, the block NB is not limited to be smaller than the traininghigh-frequency image P11 and the training medium-frequency image P12.The block NB may be as large as the training high-frequency image P11and the training medium-frequency image P12.

Processes from Steps S1002 to S1007 in FIG. 6 are performed on each ofthe blocks NB in the temporary enlarged image 121. The current block ischanged to a different block NB each time the processes from Steps S1002to S1007 are performed.

Step S1002 corresponds to the identifying, and the edge directioncalculating unit 130 detects a direction of an edge (edge direction)included in the current block. More specifically, the edge directioncalculating unit 130 extracts horizontal feature data F4 and verticalfeature data F5 by applying a horizontal high-pass filter and a verticalhigh-pass filter to the current block NB, respectively. Next, the edgedirection calculating unit 130 calculates the edge direction 131 usingthe horizontal feature data F4 and the vertical feature data F5according to the following Equation 1.Edge direction 131=arctan(F5/F4)  (Equation 1)

Here, when an absolute difference between F4 and F5 is equal to orsmaller than a predetermined threshold, the edge direction calculatingunit 130 determines the edge direction 131=an indefinite value.

For example, as illustrated in FIGS. 9A and 9B, a (first derivative)horizontal Sobel filter and a (first derivative) vertical Sobel filterare used as the horizontal high-pass filter and the vertical high-passfilter, but not limited to these.

The edge direction calculating unit 130 may calculate the edge direction131 as 3-value data based on a relationship between a threshold Th andthe difference between the horizontal feature data F4 and the verticalfeature data F5, as Equation 2 below.When(F4−F5)>Th,Edge direction 131=90°.When(F4−F5)<−Th,Edge direction 131=0°.When −Th≦(F4−F5)≦Th,Edge direction 131=an indefinite value.  (Equation 2)

The method of calculating the edge direction 131 is a typical example,and is not limited to this.

Step S1003 corresponds to the determining. At Step S1003, the regioncalculating unit 140 calculates, within the temporary enlarged image 121based on the edge direction 131, an application region to be used forsearching in the training database 110.

FIG. 10A illustrates an application region calculated for the edgedirection 131 that is vertical. FIG. 10B illustrates an applicationregion calculated for the edge direction 131 that is horizontal.

More specifically, as illustrated in FIGS. 10A and 10B, the regioncalculating unit 140 calculates an application region 122 that is arectangle having longer sides in a direction vertical to the edgedirection 131 and shorter sides in a direction horizontal to the edgedirection 131. In other words, the region calculating unit 140determines, as the shape of the application region 122, the shape havinga second width and a first width longer than the second width. Here, thefirst width is a width of the application region 122 in the directionvertical to the edge direction 131 identified at Step S1002, and thesecond width is a width of the application region 122 in the edgedirection 131. As such, since the input image 101 includes more featuresin the direction vertical to the edge direction 131 than features in theedge direction 131, the super-resolution effect with higher precisioncan be obtained in any edge direction by determining the shape havingthe second width and the first width longer than the second width, asthe shape of the application region 122.

Furthermore, the region calculating unit 140 determines, for each blockas the shape of the application region 122 for the block, a shape havingthe first width and the second width. Here, the first width is longerthan a width of the block in a direction vertical to an edge directionidentified for the block, and the second width is shorter than a widthof the block in the edge direction. More specifically, the regioncalculating unit 140 determines the shape of the application region 122so that the number of pixels included in the application region 122corresponding to the block is equal to or smaller than the number ofpixels included in the block. Accordingly, since the application region122 has a size identical to or smaller than the block corresponding tothe application region 122, the searching load can be more reduced thanthe case of searching the training medium-frequency image P12 (similarimage) that is similar to the image in the block.

Furthermore, when the edge direction 131 is determined as an indefinitevalue, the region calculating unit 140 calculates, as an applicationregion, a region that is identical to the current block or a regionhaving the different size from the current block and having the sameshape as the current block.

Furthermore, the region calculating unit 140 outputs region coordinateinformation 141 that is information for identifying the calculated ordetermined application region 122, to the feature data extracting unit150. The region coordinate information 141 indicates, for example, thecoordinates, shape, and size of the application region 122.

The application region 122 is not limited to a rectangular block. Forexample, the application region 122 may be elliptic. The method ofcalculating the application region is a typical example, and is notlimited to this.

Step S1004 corresponds to the extracting. At Step S1004, the featuredata extracting unit 150 extracts the region feature data from theapplication region 122 that is within the temporary enlarged image 121and is identified by the region coordinate information 141. Morespecifically, the feature data extracting unit 150 extracts an image ofmedium-frequency components in the application region 122 as the regionfeature data 151, by high-pass filtering the application region 122 bylinear filtering and others. In other words, the region feature data 151is an image of medium-frequency components that have been high-passfiltered, from among the frequency components of the application region122. In other words, the region feature data 151 is an image ofmedium-frequency components in the application region 122, and an imagefrom which low-frequency components in the application region 122 aredeleted. In this case, the region feature data 151 indicates pixelvalues.

Here, at least a part of the frequency band of the medium-frequencycomponents corresponding to the region feature data 151 overlaps withthe frequency band of a frequency component corresponding to thetraining medium-frequency image P12. Furthermore, the region featuredata 151 is data to be used for searching in the training database 110.

Here, the application region 122 to be processed at Step S1004 is notlimited to a region within the temporary enlarged image 121. Forexample, the application region 122 to be processed at Step S1004 may bethe current block in the input image 101.

Furthermore, the region feature data 151 is not limited to data obtainedusing a high-pass filter. The region feature data 151 may be, forexample, luminance, chrominance, and RGB values in the applicationregion 122. Furthermore, the region feature data 151 may be, forexample, moment (average, variance, kurtosis, or skewness) obtained fromthe application region 122.

Furthermore, the region feature data 151 may be, for example, a featurevalue obtained from a co-occurrence matrix (homogeneity, heterogeneity,contrast, average, standard deviation, angular second moment, orentropy). Furthermore, the region feature data 151 may be, for example,a principal component obtained by the principal component analysis and aprincipal component obtained by the independent component analysis.

The region feature data 151 is a typical example, and is not limited tothis.

Step S1005 includes part of the searching and the generating. At StepS1005, the training data search unit 160 calculates a similarity betweenthe region feature data 151 and each of the K training medium-frequencyimages P12 stored in the training database 110. The training data searchunit 160 calculates the similarity between the region feature data 151(application region 122) and each of the training medium-frequencyimages P12 only using a region having the same shape and the same areaas those of the region feature data 151.

FIG. 11 illustrates processing performed by the training data searchunit 160.

As illustrated in FIG. 10A, when the edge direction 131 of the currentblock NB [nm] is 90° (vertical direction), the application region 122 isa rectangle having the longer sides in the horizontal direction. Here,the number of pixels in the rectangular application region 122 is 18pixels×8 pixels (=144 pixels), which is the same as 12 pixels×12 pixels(=144 pixels) indicating the number of pixels in the current block NB[nm]. Here, the training data search unit 160 uses a comparison targetregion 161 having the same shape and the same area as those of theapplication region 122 in each of the training medium-frequency imagesP12 in order to calculate a similarity with the region feature data 151as illustrated in FIG. 11. The center of gravity of the comparisontarget region 161 is identical to that of each of the trainingmedium-frequency images P12.

The training data search unit 160 calculates the similarity bycalculating, for example, a sum of absolute differences (Manhattandistances) between pixel values indicated by the region feature data 151and pixel values indicated by the comparison target region 161 in eachof the training medium-frequency images P12. As the sum of absolutedifferences is smaller, a similarity between the region feature data 151and each of the training medium-frequency images P12 is higher. Here,the similarity may be calculated using the difference square sum (theEuclidean distance).

Then, the training data search unit 160 selects top L trainingmedium-frequency images P12 having the higher similarity to the regionfeature data 151, based on the calculated similarities, where L is aninteger equal to or larger than 2. L is an integer that satisfies arelational expression of 2≦L≦K. L is also a predetermined constant. InEmbodiment 1, L is assumed to be 32, for example.

The selected L training medium-frequency images P12 include a trainingmedium-frequency image P12 having the highest similarity to the regionfeature data 151. In other words, L similarities correspondingone-to-one to the selected L training medium-frequency images P12 aretop L similarities ranked in descending order of corresponding valuesfrom among all the similarities calculated at Step S1005.

In the searching included in Step S1005, the training data search unit160 searches the training database 110 for a plurality of the trainingmedium-frequency images P12 each having the image similar to the imagewithin the application region 122 having the shape determined for thecurrent block, as similar images.

Then, the training data search unit 160 selects the L traininghigh-frequency images P11 corresponding to the selected L trainingmedium-frequency images P12, from among the K training high-frequencyimages P11 stored in the training database 110. In other words, thetraining data search unit 160 selects the L training high-frequencyimages P11 corresponding to the selected L training medium-frequencyimages P12 similar to the region feature data indicating the features ofthe current block within the temporary enlarged image 121, from amongthe K training high-frequency images P11 stored in the training database110. In other words, the training data search unit 160 selects thetraining high-frequency images P11 (application images) associatedone-to-one with the training medium-frequency images P12 obtained insearching (similar images) in the training database 110, using thetraining medium-frequency images P12 (similar images) as the part of thegenerating included in Step S1005.

Then, the training data search unit 160 transmits the selected Ltraining high-frequency images P11 to the synthesizing unit 170.

In other words, at Step S1005, the training data search unit 160calculates a similarity between the region feature data 151 and each ofthe K training medium-frequency images P12, and selects the L traininghigh-frequency images P11 corresponding to the top L trainingmedium-frequency images P12 having the higher similarities to the regionfeature data 151.

The present invention is not limited to the use of the traininghigh-frequency images P11 indicating the high-frequency componentsextracted by a high-pass filter. Instead, luminance, chrominance, or RGBvalues of the training images P1 may be used.

The present invention is not limited to the use of the trainingmedium-frequency images P12 indicating the medium-frequency componentsextracted by a low-pass filter and a high-pass filter. Instead, forexample, luminance, chrominance, or RGB values of the training images P1or moment (average, variance, kurtosis, or skewness) obtained from thetraining images P1 may be used. Alternatively, instead of the trainingmedium-frequency images P12, a feature value obtained from aco-occurrence matrix (homogeneity, heterogeneity, contrast, average,standard deviation, angular second moment, or entropy) or a principalcomponent obtained by the principal component analysis and a principalcomponent obtained by the independent component analysis may be used.

Step S1006 includes the part of the generating. At Steps S1006, thesynthesizing unit 170 generates the synthesized image 171 bysynthesizing the selected L training high-frequency images P11(application images).

Step S1007 includes part of the generating. At Step S1007, asillustrated in FIG. 7, the addition unit 180 adds the synthesized image171 to the block NB (current block) corresponding to the synthesizedimage 171, in the temporary enlarged image 121. Accordingly, theresolution conversion process is performed on the current block. Thesynthesized image 171 is an image of a definition higher than that ofthe current block.

Hereinafter, the process of adding the synthesized image 171 will bedescribed in detail by giving one example. For example, assume that thecurrent block is the block NB [11] in the temporary enlarged image 121in FIG. 8. Here, the synthesizing unit 170 adds pixel values of pixelsincluded in the synthesized image 171, one-to-one to pixel values ofpixels included in the block NB [11], so that the center of thesynthesized image 171 matches the center of the block NB [11].

The processes from Step S1002 to S1007 are performed on all the blocksNB within the temporary enlarged image 121. Accordingly, the outputimage 102 is generated. In other words, with iterations of the processat Step S1007, the addition unit 180 adds the synthesized image 171 tothe temporary enlarged image 121 to generate the output image 102. Inother words, the addition unit 180 generates the output image 102 usingthe synthesized image 171.

Furthermore, with iterations of the process at Step S1007, thesynthesized images 171 are added one-to-one to adjacent blocks NB in thetemporary enlarged image 121, as illustrated in FIG. 8. Since thesynthesized images 171 are larger than the respective blocks NB, pixelsof the two synthesized images 171 are added to pixels of the blocks NBnear the boundary between the two adjacent blocks NB. Here, the adjacentblocks NB are, for example, the blocks NB [11] and [12], and blocks NB[21] and [22]. In other words, the training high-frequency images P11(application images) larger in size than the respective blocks NB andthe training medium-frequency images P12 (search images) are stored inthe training database 110 in association with each other according toEmbodiment 1. In the generating, the adding unit 180 adds thesynthesized image 171, to the current block NB and a part of at leastone of the blocks NB around the current block NB so that the center ofthe synthesized image 171 larger than the current block NB matches thecenter of the current block NB. Accordingly, artifacts between theblocks in the output image 102 can be reduced.

The image processing described above is processing when the input image101 to be processed is a still image. When the input image 101 is amoving image, the image processing is repeatedly performed for eachframe included in the moving image.

Accordingly, the image processor 100 according to Embodiment 1 cangenerate the output image 102 having a higher resolution in any edgedirection, without adding enormous processing capacity.

Since the shape of the application region 122 is not fixed and isdetermined according to an edge direction in Embodiment 1, an image inthe application region 122 in any direction of an edge can be used. Theapplication region 122 sufficiently includes features of the edge.Furthermore, since a similar image is searched for and the resolutionconversion process is performed, using the image in the applicationregion 122 having the determined shape, the super-resolution effect canbe obtained equivalently for an edge in any direction. As a result, itis possible to suppress non-uniformity in the super-resolution effectfor each block, and appropriately increase the resolution of the inputimage 101.

Although the synthesized image 171 generated by synthesizing theselected training high-frequency images P11 is added to the targetcurrent block in Embodiment 1, one of the selected traininghigh-frequency images P11 may be added to the current block withoutgenerating the synthesized image 171. In other words, in the searchingincluded at Step S1005, the training data search unit 160 searches thetraining medium-frequency images P12 (search images) held in thetraining database 110 for the training medium-frequency image P12 havingthe image similar to the image within the application region 122 havingthe shape determined for the current block, as a similar image. Then, inthe generating from Steps S1005 to S1007, the training data search unit160 selects the training high-frequency image P11 (application image)associated with the similar image obtained in the searching, from thetraining database 110 using the similar image. Furthermore, in thegenerating, the adding unit 180 adds the selected traininghigh-frequency image P11 (application image) to the current block so asto perform the resolution conversion process on the current block.

Embodiment 2

Next, an image processor according to Embodiment 2 of the presentinvention will be described.

FIG. 12 is a block diagram illustrating a configuration of an imageprocessor 500 according to Embodiment 2 of the present invention. Theimage processor 500 according to Embodiment 2 generates an output image102 having a resolution higher than that of an input image 101 with thesuper-resolution technique using the movement of images, that is, withthe position matching in sub-pixel precision and pixel interpolation,without using the training database 110.

The image processor 500 receives a decoded moving image MV10 from, forexample, an external decoder that is not illustrated. The moving imageMV10 is, for example, a moving image decoded in accordance with theH.264/AVC standard. The moving image MV10 is not limited to the movingimage in accordance with the H.264/AVC standard, and may be a movingimage decoded, for example, in accordance with the MPEG-2 standard.

The moving image MV10 is assumed to be, for example, a moving imagecorresponding to one Group Of Pictures (GOP). The moving image MV10includes pictures. The image processor 500 receives, for example, afirst picture in display order as the input image 101 from among Qpictures included in the moving image MV10, where Q is an integer equalto or larger than 2. In Embodiment 2, the input image 101 is a stillimage. In Embodiment 2, Q is assumed to be 10, for example.

Although the details will be described later, according to Embodiment 2,a position of an application region corresponding to each block includedin the low-resolution input image 101 is matched to a position of acorresponding one of L low-resolution reference images, andinterpolation pixel values are estimated based on a result of thematching so as to generate the output image 102. Here, L denotes apredetermined fixed number. Each of the L low-resolution referenceimages is different from the input image 101 and is a picture includedin the Q pictures.

As illustrated in FIG. 12, the image processor 500 includes an edgedirection calculating unit 130, a region calculating unit 140, aposition matching unit 520, and a high-resolution image generating unit530.

The edge direction calculating unit 130 calculates an edge direction 131for each of the blocks included in the input image 101 in the samemanner as Embodiment 1. The region calculating unit 140 calculates, foreach of the blocks included in the input image 101, an applicationregion 122 corresponding to the block, in the edge direction 131calculated for the block. Furthermore, the region calculating unit 140outputs region coordinate information 141 that is information foridentifying the application region 122.

The position matching unit 520 receives the moving image MV10 includingthe Q pictures from, for example, the external decoder that is notillustrated. The position matching unit 520 performs a position matchingprocess. The details will be described later. The position matching unit520 matches the position of the application region 122 calculated forthe current block, to a position of a corresponding one of the Lreference images. In other words, the position matching unit 520 matchesa position of a picture other than the input image 101, among the movingimage MV10 that include the input image 101 and pictures, to theposition of the image in the application region 122 having thedetermined shape to search for a similar image corresponding to thecurrent block. The position matching is performed with, for example,sub-pixel precision.

FIG. 13 is a flowchart of the image processing performed by the imageprocessor 500 according to Embodiment 2.

The input image 101 is partitioned into blocks. Hereinafter, the currentblock to be processed among blocks included in the input image 101 willalso be referred to as a current block PB. The current block PB has, forexample, a size of horizontal 8 pixels×vertical 8 pixels.

At Step S1002, the edge direction calculating unit 130 calculates theedge direction 131 that is a direction of an edge included in thecurrent block PB as in Embodiment 1. Then, the edge directioncalculating unit 130 transmits the calculated edge direction 131 to theregion calculating unit 140.

At Step S1003, the region calculating unit 140 calculates theapplication region 122 included in the input image 101 and correspondingto the current block PB, based on the edge direction 131 of the currentblock PB as in Embodiment 1. Furthermore, the region calculating unit140 outputs, to the position matching unit 520, the region coordinateinformation 141 that is information for identifying the applicationregion 122.

Step S3003 corresponds to the searching according to Embodiment 2, andthe position matching process is performed at Step S3003. In theposition matching process, the position matching unit 520 matches theposition of the application region 122 that is included in the inputimage 101 and is identified by the region coordinate information 141, tothe position of each of the L reference images.

More specifically, the position matching unit 520 determines, forexample, the second to (L+1)-th pictures in display order from among theQ pictures as reference images. Embodiment 2 is described assuming L=4.

FIG. 14 illustrates the position matching process.

FIG. 14 exemplifies pictures PC[1], PC[2], PC[3], PC[4], and PC[5]. Thepictures PC[1] to PC[5] respectively correspond to the first to fifthpictures.

The picture PC[1] is the input image 101. In the picture PC[1], thecurrent block PB and the application region 122 calculated for thecurrent block PB are indicated. The pictures PC[2] to PC[5] arereference images. The reference images are searched for respectivesimilar images 122 a to 122 d that are similar to the image in theapplication region 122 in the position matching process. A block RB [n(integer)] in each of the reference images is a reference blockcorresponding to a corresponding one of the similar images 122 a to 122d.

In the position matching process, the position matching unit 520 matchesthe position of the current block PB to the position of each of thepictures PC[2], PC[3], PC[4], and PC[5]. For example, the positionmatching unit 520 interpolates pixels in each of the reference images,by calculating pixel values of sub-pixels of each of the referenceimages. Then, the position matching unit 520 detects (searches), fromthe respective reference images, the similar images 122 a to 122 d thatare similar to the image in the application region 122 with thesub-pixel precision. For example, the position matching unit 520 detectsthe similar image so that a sum of absolute differences between eachpixel value of the application region 122 and a pixel value of thecorresponding similar image is the smallest. Furthermore, the positionmatching unit 520 detects displacement in a position each between theapplication region 122 and two of the similar images 122 a to 122 d, asa displacement amount (motion vector) 521 with sub-pixel precision.Then, the position matching unit 520 outputs, to the high-resolutionimage generating unit 530, (i) the displacement amount 521 detected foreach of the reference images and (ii) the reference image as referencedata 522.

Step S3004 corresponds to the generating according to Embodiment 2, andthe pixel interpolating process is performed at Step S3004. In the pixelinterpolating process, the high-resolution image generating unit 530extracts or obtains, from the reference data 522 (reference images), thereference block RB at a position displaced from the current block PB bythe displacement amount 521 detected for each of the reference images.Then, the high-resolution image generating unit 530 performs theresolution conversion process on the current block PB by interpolatingthe pixels in the current block PB using the pixels of the referenceblocks RB[2] to RB[5] extracted from the reference images.

The processes from Step S1002, S1003, S3003, and S3004 are performed onall the blocks within the input image 101. Accordingly, the output image102 is generated. In other words, with iterations of the process at StepS3004, the high-resolution image generating unit 530 generates theoutput image 102 having a resolution higher than that of the input image101.

The processes from Step S1002, S1003, S3003, and S3004 may be performedon, not limited to all the blocks in the input image 101, a blockdesignated in the input image 101.

Since in Embodiment 2, the shape of the application region 122 is notfixed and is determined according to an edge direction as in Embodiment1, the super-resolution effect can be obtained equivalently for an edgein any direction. As a result, it is possible to suppress non-uniformityin the super-resolution effect for each block, and appropriatelyincrease the resolution of the input image 101. Furthermore, the similarimages for the current block PB are searched for by matching theposition of the picture other than the input image to the position ofthe image within the application region 122 with the sub-pixelprecision, that is, by performing the motion estimation and the motioncompensation with the sub-pixel precision. Thus, it is possible toappropriately increase the resolution of the moving image.

Embodiment 3

The processing described in each of Embodiments can be easilyimplemented by an independent computer system, by recording a programfor realizing the image processing method described in Embodiment on arecording medium such as a flexible disk.

FIGS. 15A to 15C are diagrams explaining a case where the imageprocessing method in each of Embodiments is implemented by a computersystem using the program recorded on a recording medium such as aflexible disk.

FIG. 15B illustrates a front appearance of the flexible disk, a crosssection of the flexible disk, and the flexible disk. FIG. 15Aillustrates an example of a physical format of the flexible disk as therecording medium body. A flexible disk FD is contained in a case F, andtracks Tr are concentrically formed on a surface of the flexible disk FDfrom outer to inner peripheries. Each track is divided into 16 sectorsSe in an angular direction. This being so, in the flexible disk FDstoring the above-mentioned program, the program is recorded in an areaallocated on the flexible disk FD.

Furthermore, FIG. 15C illustrates a configuration for recording theprogram on the flexible disk FD and reproducing the program. In the caseof recording the program for implementing the image processing method onthe flexible disk FD, the program is written from a computer system Csvia a flexible disk drive FDD. In the case of implementing the imageprocessing method on the computer system Cs by the program recorded onthe flexible disk FD, the program is read from the flexible disk FD andtransferred to the computer system Cs via the flexible disk drive FDD.

Although the above describes an example of using the flexible disk as arecording medium, an optical disc may equally be used. Moreover, therecording medium is not limited to such, and any recording medium suchas an integrated circuit (IC) card and a read-only memory (ROM) cassetteis applicable so long as the program can be recorded.

Embodiment 4

FIG. 16 illustrates a configuration of a television receiver 800according to Embodiment 4 in the present invention, using the imageprocessor and the image processing method according to each of aboveEmbodiments.

The television receiver 800 includes a broadcast reception apparatus801, an input selection apparatus 802, an image processor 803, a paneldrive apparatus 804, and a display panel 805. Although the apparatuses801 to 804 are located outside the display panel 805 in FIG. 16, theapparatuses 801 to 804 may be located inside the display panel 805.

The image processor 803 has the same functions and configuration asthose of the image processor 100 according to one of Embodiments 1 and2.

The broadcast reception apparatus 801 receives broadcast waves from anantenna output signal 821 outputted from an external antenna (notillustrated), and outputs a video signal obtained by demodulating thebroadcast waves, as a broadcast video signal 822.

The input selection apparatus 802 selects one of the broadcast videosignal 822 and an external video signal 820 that is outputted from arecorder such as a digital versatile disc (DVD) and a Blu-ray Disc (BD),or an external video appliance such as a DVD and a BD player, accordingto the user's selection. The input selection apparatus 802 outputs theselected video signal as an input video signal 823.

In the case where the input video signal 823 is an interlace signal, theimage processor 803 converts the input video signal 823 into aprogressive signal. In other words, the image processor 803 performs I/Pconversion on the input video signal 823. Furthermore, the imageprocessor 803 may perform image quality improvement processing forimproving contrast on the input video signal 823.

Furthermore, the image processor 803 performs the image processing onthe input video signal 823 according to the image processing method ofone of Embodiments 1 and 2. Then, the image processor 803 outputs aresult of the processing as a quality-improved video signal 824.

The panel drive apparatus 804 converts the quality-improved video signal824 into a dedicated signal for driving the display panel 805, andoutputs the dedicated signal as a panel drive video signal 825.

The display panel 805 converts an electrical signal into an opticalsignal according to the panel drive video signal 825, and displaysdesired video based on the obtained optical signal.

In such a way, the image processor or the image processing methodaccording to each of Embodiments is applicable to the televisionreceiver 800. This allows the television receiver 800 to achieve theadvantages described in each of Embodiments. Note that the imageprocessor or the image processing method according to each ofEmbodiments is not limited to the application to a television receiver,and may equally be used in various digital video appliances such as arecorder, a player, and a mobile appliance. In all cases, the advantagesdescribed in Embodiments can be achieved.

Examples of the recorder include a DVD recorder, a BD recorder, and ahard disk recorder. Examples of the player include a DVD player and a BDplayer. Examples of the mobile appliance include a mobile phone and aPersonal Digital Assistant (PDA).

Although the image processor and the image processing method accordingto the present invention or other aspects thereof are described based oneach of Embodiments, the present invention is not limited toEmbodiments.

For example, as illustrated in FIG. 17, the image processor according tothe present invention may include the general or comprehensiveconfiguration of Embodiments 1 and 2 as another aspect.

FIG. 17 is a block diagram illustrating a functional configuration of animage processor 900 according to another aspect of the presentinvention.

The image processor 900 is an apparatus that generates, using an inputimage, an output image having a resolution higher than a resolution ofthe input image, and includes a direction identifying unit 910, a regiondetermining unit 920, an image search unit 925, and an image generatingunit 930.

The direction identifying unit 910 identifies an edge direction that isa direction along an edge included in the input image. The directionidentifying unit 910 corresponds to the edge direction calculating unit130 according to one of Embodiments 1 and 2.

The region determining unit 920 determines the shape of an applicationregion that is a region including at least a part of an edge, accordingto the edge direction identified by the direction identifying unit 910.The region determining unit 920 corresponds to the region calculatingunit 140 according to one of Embodiments 1 and 2.

The image search unit 925 searches for an image similar to an imagewithin the application region having the shape determined by the regiondetermining unit 920. The image search unit 925 is implemented by a partof the functional blocks included in the training data search unit 160according to Embodiment 1. In other words, the image search unit 925corresponds to a part of the functional blocks included in the trainingdata search unit 160 that searches the training database 110 for aplurality of the training medium-frequency images P12 similar to theregion feature data 151 of the application region 122. Alternatively,the image search unit 925 is implemented by part of the functionalblocks included in the position matching unit 520 according toEmbodiment 2. In other words, the image search unit 925 corresponds topart of the functional blocks included in the position matching unit 520that searches a plurality of reference images for similar images similarto the image of the application region 122 by performing a positionmatching process.

The image generating unit 930 generates the output image by performingthe resolution conversion process on the input image, using the similarimage so that the input image includes high-frequency components. Theimage generating unit 930 is implemented by the training database 110,part of functional blocks of the training data search unit 160, andfunctional blocks including the synthesizing unit 170 and the addingunit 180, according to Embodiment 1. Furthermore, the image generatingunit 930 is implemented by the high-resolution image generating unit 530according to Embodiment 2.

Furthermore, the image processing method corresponding to the imageprocessor 900 includes, as steps, the processes performed by thedirection identifying unit 910, the region determining unit 920, theimage search unit 925, and the image generating unit 930.

Part or all of the direction identifying unit 910, the regiondetermining unit 920, the image search unit 925, and the imagegenerating unit 930 included in the image processor 900 may be includedin hardware, such as a system large scale integration (LSI).Furthermore, part or all of the direction identifying unit 910, theregion determining unit 920, the image search unit 925, and the imagegenerating unit 930 may be a module of a program executed by aprocessor, such as a central processing unit (CPU). Furthermore, theimage processor 900 may be implemented as an integrated circuit.

For example, the size of the training high-frequency images P11 may bedifferent from the size of the training medium-frequency images P12,where the training medium-frequency images P12 and the traininghigh-frequency images P11 are stored in the training database 110.Furthermore, in this case, the training high-frequency images P11 may besearched for (selected) using, for example, the input image 101 insteadof the temporary enlarged image 121 according to Embodiment 1.

Furthermore, obviously, the present invention includes, for example, anoptical disc recording apparatus, a moving image transmitting apparatus,a digital television broadcasting apparatus, a Web server, acommunication apparatus, and a mobile information terminal each of whichincludes the image processor according to each of Embodiments.Similarly, the present invention obviously includes a moving imagereceiving apparatus, a moving image recording apparatus, a still imagerecording apparatus, a digital television broadcast receiving apparatus,a communication apparatus, and a mobile information terminal each ofwhich includes the image processor according to each of Embodiments.Here, examples of the moving image recording apparatus include acamcorder and a web server, and examples of the still image recordingapparatus include a digital still camera.

The values used in all Embodiments are examples of values forspecifically describing the present invention. In other words, thepresent invention is not limited to each of the values used inEmbodiments.

Furthermore, the image processing method according to the presentinvention does not necessarily have to include all the steps in FIG. 6.In other words, the image processing method according to the presentinvention has only to include steps as little as possible to produce theadvantages of the present invention.

Furthermore, the order for performing the steps included in the imageprocessing method according to each of Embodiments is an example forspecifically describing the present invention, and the other orders maybe used. Furthermore, part of the steps in the image processing methodmay be performed in parallel with and separately from the other steps.For example, in FIG. 6, Steps S1002 and 1003 may be performed inparallel with and separately from Steps S1004 and 1005.

Part or all of the functional patches (constituent elements) of each ofthe image processors 100 to 500 are typically implemented as a systemlarge scale integration (LSI) that is an integrated circuit. Thesepatches can be separately made in a plurality of single-function LSIs,or also can be made in one integrated LSI to include part or all of thepatches.

The name used here is LSI, but it may also be called IC, system LSI,super LSI, or ultra LSI depending on the degree of integration.

Moreover, not only the LSI, but also a dedicated circuit or a generalpurpose processor and so forth can also achieve the integration. FieldProgrammable Gate Array (FPGA) that can be programmed aftermanufacturing LSI or a reconfigurable processor that allowsre-configuration of the connection or configuration of LSI can be usedfor the same purpose.

Moreover, with advancement in semiconductor technology and appearance ofderivatives of the technology that may replace LSI, the functionalpatches may be integrated using that technology. Application ofbiotechnology is one such possibility.

Furthermore, among these patches (constituent elements), a portion forstoring data to be searched for can be differently configured withoutbeing made in one integrated LSI.

Each of the constituent elements may be implemented by dedicatedhardware or executing a software program suitable for the constituentelement. Each of the constituent elements may be implemented by readinga software program recorded on a recording medium, such as a hard diskor a semiconductor memory, using a program executing unit, such as a CPUand a processor. The software that implements the image processoraccording to each of Embodiments is the following program.

In other words, the program causes a computer to execute: identifying anedge direction that is a direction along an edge included in the inputimage; determining a shape of an application region according to theidentified edge direction, the application region being a regionincluding at least a part of the edge; searching for an image similar toan image within the application region having the determined shape; andgenerating the output image by performing a resolution conversionprocess on the input image using the similar image so that the inputimage includes high-frequency components.

Although the image processor and the image processing method accordingto one or more aspects of the present invention are described based oneach of Embodiments, the present invention is not limited toEmbodiments. Without departing from the scope of the present invention,the one or more aspects of the present invention may include anembodiment with some modifications conceived by a person skilled in theart on each of Embodiment, and an embodiment by combining theconstituent elements included in different Embodiments.

Embodiments disclosed this time are exemplifications in all respects,and should be regarded as not limiting the scope of the presentinvention. The scope of the present invention is indicated not by thedescription but by Claims, and is intended to include all themodifications within Claims, meanings of equivalents, and the scope ofthe equivalents.

The image processing method and the image processor according to thepresent disclosure enable generation of a high-resolution image withhigher definition, and is applicable to, for example, a camcorder and adigital still camera.

The invention claimed is:
 1. An image processing method of generating,using an input image, an output image having a resolution higher than aresolution of the input image, the method comprising: identifying anedge direction that is a direction along an edge included in the inputimage; determining a shape of an application region according to theidentified edge direction, the application region being a regionincluding at least a part of the edge; searching for an image similar toan image within the application region having the determined shape; andgenerating the output image by performing a resolution conversionprocess on the input image using the similar image so that the inputimage includes high-frequency components, wherein in the determining ofthe shape, a shape having a second width and a first width longer thanthe second width is determined as the shape of the application region,the second width being a width of the application region in theidentified edge direction and the first width being a width of theapplication region in a direction vertical to the edge direction.
 2. Theimage processing method according to claim 1, wherein in theidentifying, an edge direction of an edge is identified for each ofblocks included in the input image, the edge being included in theblock, in the determining, a shape of an application region isdetermined for each of the blocks, according to the edge directionidentified for the block, in the searching, an image similar to an imagewithin the application region having the shape determined for the blockis searched for each of the blocks, and in the generating, the outputimage is generated by performing the resolution conversion process oneach of the blocks using the similar image that is searched for theblock.
 3. The image processing method according to claim 2, wherein inthe determining, a shape having the first width and the second width isdetermined for each of the blocks as the shape of the application regionfor the block, the first width being longer than a width of the block ina direction vertical to the edge direction, the second width beingshorter than a width of the block in the edge direction, and the edgedirection being identified for the block.
 4. The image processing methodaccording to claim 3, wherein in the determining, the shape of theapplication region is determined so that the number of pixels includedin the application region corresponding to each of the blocks is equalto or smaller than the number of pixels included in the block.
 5. Theimage processing method according to claim 2, wherein in the searching,(i) a database is used, the database holding a plurality of searchimages, and a plurality of application images associated one-to-one withthe search images and including more high-frequency components thanhigh-frequency components of the search images, and (ii) the searchimages held in the database are searched for, as the similar image, asearch image including an image similar to an image within anapplication region having a shape determined for a current block to beprocessed, and in the generating, the resolution conversion process isperformed on the current block by selecting, from the database using thesimilar image obtained in the searching, an application image associatedwith the similar image and adding the selected application image to thecurrent block.
 6. The image processing method according to claim 5,wherein in the searching, the database is searched for a plurality ofsearch images as a plurality of similar images, the search images eachincluding an image similar to the image within the application regionhaving the shape determined for the current block, and in thegenerating, the resolution conversion process is performed on thecurrent block by (i) selecting, from the database using the similarimages obtained in the searching, a plurality of application imagesassociated one-to-one with the similar images, (ii) generating asynthesized image by synthesizing the selected application images, and(iii) adding the synthesized image to the current block.
 7. The imageprocessing method according to claim 6, wherein a plurality ofapplication images larger in size than the blocks are held in thedatabase in one-to-one association with the search images, and in thegenerating, the synthesized image is added to the current block and apart of at least one of the blocks around the current block so that thecenter of the synthesized image matches the center of the current block,the synthesized image being larger than the current block.
 8. The imageprocessing method according to claim 5, further comprising enlarging theinput image, wherein the identifying, the determining, the searching,and the generating are performed on each of blocks included in theenlarged input image.
 9. The image processing method according to claim8, further comprising extracting, as medium-frequency components fromeach of the blocks included in the enlarged input image, frequencycomponents excluding low-frequency components, wherein in the searching,a search image is searched for each of the blocks, the search imageincluding an image similar to an image of the medium-frequencycomponents within an application region having a shape determined forthe block.
 10. The image processing method according to claim 2, whereinin the searching, a similar image corresponding to a current block to beprocessed is searched for by matching a position of the image within theapplication region having the determined shape to a position of apicture included in a moving image, the picture being other than theinput image, and the moving image including the input image that is apicture, and in the generating, a reference block including at least apart of the similar image obtained in the searching is obtained from thepicture other than the input image using the similar image, and theresolution conversion process is performed on the current block byinterpolating pixels in the current block using pixels in the referenceblock.
 11. An image processor that generates, using an input image, anoutput image having a resolution higher than a resolution of the inputimage, the image processor comprising: a direction identifying unitconfigured to identify an edge direction that is a direction along anedge included in the input image; a region determining unit configuredto determine a shape of an application region according to theidentified edge direction, the application region being a regionincluding at least a part of the edge; an image search unit configuredto search for an image similar to an image within the application regionhaving the determined shape; and an image generating unit configured togenerate the output image by performing a resolution conversion processon the input image using the similar image so that the input imageincludes high-frequency components, wherein a shape having a secondwidth and a first width longer than the second width is determined asthe shape of the application region determined by the determining unit,the second width being a width of the application region in theidentified edge direction and the first width being a width of theapplication region in a direction vertical to the edge direction.
 12. Anintegrated circuit that generates, using an input image, an output imagehaving a resolution higher than a resolution of the input image, theintegrated circuit comprising: a direction identifying unit configuredto identify an edge direction that is a direction along an edge includedin the input image; a region determining unit configured to determine ashape of an application region according to the identified edgedirection, the application region being a region including at least apart of the edge; an image search unit configured to search for an imagesimilar to an image within the application region having the determinedshape; and an image generating unit configured to generate the outputimage by performing a resolution conversion process on the input imageusing the similar image so that the input image includes high-frequencycomponents, wherein a shape having a second width and a first widthlonger than the second width is determined as the shape of theapplication region determined by the determining unit, the second widthbeing a width of the application region in the identified edge directionand the first width being a width of the application region in adirection vertical to the edge direction.
 13. A non-transitorycomputer-readable recording medium on which a program for generating,using an input image, an output image having a resolution higher than aresolution of the input image is recorded, the program causing acomputer to execute: identifying an edge direction that is a directionalong an edge included in the input image; determining a shape of anapplication region according to the identified edge direction, theapplication region being a region including at least a part of the edge;searching for an image similar to an image within the application regionhaving the determined shape; and generating the output image byperforming a resolution conversion process on the input image using thesimilar image so that the input image includes high-frequencycomponents, wherein in the determining of the shape, a shape having asecond width and a first width longer than the second width isdetermined as the shape of the application region, the second widthbeing a width of the application region in the identified edgedirection, and the first width being a width of the application regionin a direction vertical to the edge direction.